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(57) Abstract: A method for reconstructing a digitized image, comprising the steps of: retrieving from a memory a digitized image 
JlJ of an off-axis hologram, an off-axis interferogram or a binned image originating with a digital camera, the digitized image having 
parameters defining a position of a sideband of the image in Fourier space; applying the Extended Fourier transform to the image 
to directly and correctly center the sideband; and, displaying a resulting image in phase and/or amplitude. The method can further 
comprise the steps of: first digitizing the image; detennining the parameters; storing the parameters in the memory; and binning the 
image after retrieving it from the camera and/or memory; or applying an aperture to limit the sideband. 
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HIGH SPEED EVALUATION OF DIGITIZED IMAGES 

Statement Regarding Federally-Sponsored 
Research or Development 

This invention was made with government support under contract 
5 DE-AC05-96OR22464, awarded by the United States Department of Energy to Lockheed 

Martin Energy Research Corporation, and the United States Government has certain rights in 
this invention. 

Background of the Invention 

1. Field of the Invention 

10 This invention relates to the field of processing digital images, and in particular, to 

reconstructing digital images of holograms, interferograms and binned images thereof with 
phase and amplitude information. 

2. Description of Related Art 

When recording an image wave, the information about the phase of the image wave, 
1 5 which is part of the complex image wave, is destroyed and only the absolute square of the 
image wave is recorded, yielding the final image. When recording off-axis holograms or 
interferograms of arbitrary wavelength and independent of the nature of the waves, the 
information on the phase of the image wave, that is the image phase, is preserved. However, 
the hologram or interferogram has to be processed before the phase information is accessible. 
20 Historically, holograms or interferograms have been and still are, in most laboratories, 

recorded on film and then processed on an optical bench to make the phase information 
visible. The more modern approach is to record and process holograms or interferograms 
digitally. The standard digital processing (see J. F. Frank, K.-H Herrmann and H. Lichte, 
"Off-Axis Electron Holography With Digital Object Reconstruction", Proc. 11th International 
25 Congress on Electron Microscopy , Kyoto., 1986, pp. 677-678) includes two Fourier 

transforms and the atan2 function. The first Fourier transform slows down the processing 
significantly and presently prohibits video frame phase image display (see note on 
reconstruction time in E. Voelkl, L. F. Allard, B. Rost, "A Software Package for the 
Processing and Reconstruction of Electron Holograms", Journal of Microscopy , Vol. 180, 
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Pt. 1, October 1995, pp 39-50). An effort to circumvent this problem (see J. Chen, T. 

Hirayama, C. Lai, T. Tanji, K. Ishizuka, A. Tonomura, "Real-time Electron Holography 

Using a Liquid-crystal Panel", in Electron Holography . Eds. A. Tonomura, L.F. Allard, G. 

Pozzi, D.C. Joy, Y. A. Ono, 1995 Elsevier Science B.V.) feeds digital images back into an 
5 optical system to perform the reconstruction algorithm optically, then re-records the resulting 

image, which corresponds directly to the image phase, for display on a screen. 

The Extended Fourier transform is described in E. Voelkl, L. F. Allard, A. Datye, B. 

Frost, "Advanced Electron Holography: A New Algorithm for Image Processing and a 

Standardized Quality Test for the Feg Electron Microscope", Ultramicroscopv. Vol. 58, 1995, 
10 pp 97-103. A time estimate for the standard fully digital reconstruction of holograms is given 

in E. Voelkl, L. F. Allard, B. Rost, "A Software Package for the Processing and 

Reconstruction of Electron Holograms". Journal of Microscopy . Vol. 180, Pt. 1, October 

1995, pp 39-50. Papers relevant to digital sampling of holograms include, for example: K. 

Ishizuka, "Number of Sampling Points Required for Aberration Correction Using Off-Axis 
1 5 Electron Holography", Ultramicroscopv. Vol. 53, 1994, pp. 297-303; and, K. Ishizuka, 

"Optimized Sampling Schemes for Off-axis Holography", Ultramicroscopv . Vol. 52, 1993, 

pp. 1-5. 

Many modern digital cameras allow binning of the image data by the camera 
hardware, where binning takes either no time or insignificant time. In fact, the readout time 

20 of an image from a camera with hardware binning is significantly shortened, allowing for 
more frames per second for binned images than can be reached for non-binned images. 
Binning of an image, for example by a factor of two, means that a 2 x 2 pixel area is treated 
as one pixel. Thus, a 1024 x 1024 pixel image shrinks to 512 x 512 pixels, or 1/4 of the 
original number of pixels. 

25 For any holography or interferometry setup, the consequences of binning are that the 

sampling frequency of the carrier frequency falls either very close to, or even beyond, the 
Nyquist limit. Holograms or interferograms taken with carrier frequencies close to, at or 
beyond the Nyquist limit have been discussed in the literature only recently. The fact that 
holograms or interferograms can be recorded and processed successfully at or slightly beyond 

30 the Nyquist limit has been discussed in detail in K. Ishizuka, "Optimized Sampling Schemes 
For Off- Axis Holography", Ultramicroscopv , Vol. 52, 1993, pp. 1-5. Holograms and 
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interferograms with a carrier frequency recorded beyond the Nyquist limit are demonstrated 
and discussed in E. Voelkl, L. F. Allard, B. Frost, "Recording, Display and Evaluation 
Methods to Obtain Quantitative Information from Electron Holograms", Journal Materials 
Characterization , published by Elsevier Science, in press) and are not relevant to the 
5 inventive arrangements taught herein. 
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Summary of the Invention 
This invention is based on full digital processing of holograms, interferograms and 
binned images thereof, and goes beyond the published procedures and known prior art in two 
areas. In accordance with the inventive arrangements, an Extended Fourier Transform (EFT) 
5 is utilized, and moreover, processing can be enhanced by digital sub-Nyquist recording of the 
off-axis hologram or interferogram. Each of these steps greatly reduces the time required for 
a full reconstruction of the phase and amplitude image and reduces the amount of memory 
necessary for the reconstruction process. The reduction in memory requirement is significant, 
insofar as it allows standard number crunching boards, for example quadputers (see 
10 http://www.microway.com/history/quad.html) to be utilized even for images larger than 512 
by 512 pixels. 

The use of an EFT for processing digital images, as explained briefly above, has not 
been discussed in the literature or known prior art in the context of recording holograms and 
interferograms close to or below the Nyquist limit. 

1 5 The inventive arrangements make it possible to greatly accelerate the process of 

reconstructing the phase and amplitude (modulus) information present in a digital hologram 
or interferogram or binned version thereof and to reduce the amount of memory required for 
the reconstruction process. 

A method for reconstructing a digitized image, in accordance with the inventive 

20 arrangements comprising the steps of: retrieving from memory a digitized image and 

parameters defining a position of a sideband of the image in Fourier space; applying the 
Extended Fourier transform to the image to directly and correctly center and minimize the 
sideband; applying an inverse Fourier transform to the sideband; and, displaying a resulting 
image in at least one of phase and amplitude. Optionally, the method can include applying an 

25 aperture to the sideband. E. Voelkl, H. Lichte, "Electron Holograms for subangstrom point 
resolution," Ultramicroscopy, 32(1990)177-180. 

The method can comprise the step of retrieving from the memory a digitized image of 
an off-axis hologram, from an off-axis interferogram, or a binned image thereof. 

The method can further comprise the steps of: first digitizing an image of an off-axis 

30 hologram or an off-axis interferogram, digitizing the image under under-sampling conditions 
(i.e., move the position of the sideband beyond the Nyquist limit), or binn the digitized image 
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after digitizing. 

The method can comprise the step of including in the image terms representing the 
sampling frequency for the interference fringes in two dimensions, i.e., the position of the 
sideband in Fourier space, or retrieving that image function from memory. 

The steps can be undertaken independently of the nature of the wave and 
independently of wavelength. 

The steps can be implemented substantially in real-time. 
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Brief Description of the Drawings 
Figure 1 is a flowchart useful for explaining the inventive arrangements. 
Figure 2 is a flow chart useful for explaining the prior art method. 
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Detailed Description of the Preferred Embodiments 
A method for processing digital images in accordance with the inventive 
arrangements, and in particular a method for reconstructing digital images of off-axis 
holograms, interferograms and binned digital images is illustrated by flow chart 40 in Figure 
1 . The prior art method is illustrated by flow chart 10 in Figure 2. As will become apparent, 
steps 44 and 14 correspond to one another, steps 50 and 20 correspond to one another, steps 
52 and 22 correspond to one another and steps 54 and 24 correspond to one another. Step 42 
of method 40 begins with the step of storing a digitized image from an off-axis hologram, an 
off-axis interferogram or a binned image, as can be generated by a digital camera. Step 12 of 
method K) begins with the step of storing a digitized image from an off-axis hologram or an 
off-axis interferogram. The prior art has not considered the reconstruction of binned images 
in this context. 

A digitized image is retrieved from a memory in accordance with each of steps 44 and 
14, and in each case, the image being retrieved can be, for example, 1024 x 1024 pixels. 

A hologram or interferogram in a digitized form is described by I(m, n) y where m e 
{ 1 , M\ and n e { 1 , N\ are integers and M p N are the number of pixels the digitizing 
device yields in the x-direction and y-direction respectively, as 

I{m,n) = 1 + a 2 {m y n) + 2a(m,/?)iicos[2II(/w/.y w + nls n ) + Y(m,w)] (1) 

In equation (1) the term a(m,n) describes the amplitude (or modulus) information in the 
image, \i describes the contrast of the interference fringes, determined by the cosine term), 
¥(///,//) describes the phase information in the image and s /n and s it describe the number of 
pixels per interference fringe (or sampling frequency) along the x-axis and y- axis 
respectively. 

Several methods exist to extract the modulus and phase from the interferogram or hologram. 
These methods are, for example, described in: J. F. Frank, K.-H. Herrmann and H. Lichte, 
"Off-Axis Electron Holography With Digital Object Reconstruction", Proc. 11th International 
Congress on Electron Microscopy , Kyoto., 1986, pp. 677-678; E. Voelkl and H. Lichte, 
"Electron Holograms for Subangstrom Point Resolution", Ultramicroscopv . Vol. 32, 1990, 
pp. 177-180; Q. Ru, T. Hirayama, J. Endo, A. Tonomura, "Hologram-Shifting Method For 
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High-speed Electron Hologram Reconstruction", Jap. J. of Appl. Phvs. . Vol. 31, 1992, 
pp. 19 19-1 921; M. Lehmann, E. Voe and R Lenz, "Reconstruction Of Off-axis Holograms: A 
New And Fast Alternative Method", Ultramicroscopv. Vol. 54, 1994, pp. 335-344; W.D. 
Rau, H. Lichte, E. Voelkl and U. Weierstall, "Real-Time Reconstruction of Electron-Off- 
5 Axis Holograms Recorded with a High Pixel CCD Camera", Journal of Computer Assisted 
Microscopy . Vol. 3, No. 2, 1991; and, J. Chen, T. Hirayama, C. Lai, T. Tanji, K. Ishizuka, A. 
Tonomura, "Real-time Electron Holography Using a Liquid-crystal Panel", Electron 
Holography , Eds. A. Tonomura, L. F. Allard, G. Pozzi, D. C. Joy, Y. A. Ono, 1995 Elsevier 
Science B.V. 

10 In accordance with the inventive arrangements, and contrary to the prior art, the 

standard reconstruction method based on the standard Fourier algorithm is modified to 
accelerate the time necessary for the reconstruction process and reduce the amount of 
memory necessary for the process. In the standard reconstruction process, in accordance with 
step 16 of method JO, a forward Fourier transform (FT)is applied to I(m,n) 9 

M.N (AVA*);;i y {V+M)n 

G(*V') = EFT{/(m,A2)} = £/(/w,/2)e * N (2) 

m.n=l 

1 5 where G(k,l) is a complex function, where k,l are the discrete 

G(k,l)= FT{ /(/»,«)} = I(m,n)e M » (3) 

m, n = 1 

values in the reciprocal space, or Fourier space, and where 
k € {-M/2,...,M/2-l} and 1 € {-N/2,...,N/2-l}. 

The prior art method then requires that the sideband be isolated in accordance with 
step 1 8, involving for example 256 x 256 pixels. 
20 Contrary to the prior art, and in accordance with the inventive arrangements, instead 

of applying the standard Fourier transform, typically referred to as the fast Fourier transform 
(FFT) because of the way the standard Fourier transform is programmed, the Extended 
Fourier transform (EFT) is applied. The Extended Fourier transform is described in E. 



-8- 



WO 01/50418 



PCT/US01/00588 



Voelkl, L. F. Allard, B. Frost, "Practical Electron Holography: Applications of Advanced 
Hologram Processing Techniques to Materials Science Problems", Electron Holography . Eds. 
A. Tonomura, L. F. Allard, G. Pozzi, D. C. Joy, Y. A. Ono, 1995 Elsevier Science B.V. In 
accordance with the EFT, 



1,1 2Bi( Ai!*Ail, 2 .i|iA + iA, 

6(1 1 ') =EFT[I(a, i).] = E [I(«,n)e M 8 ]e " s (4) 

it i s i 



5 but where the range for k ' and / ' is significantly limited. 

Instead of using Equations (3) and (4), G(k\ /*) can also be approximated reasonably 
by performing a standard Fourier transform on the expression I b (/a v) 



G(k\V)=FFT{Un,v)} (5) 



where B is a binning factor (B is an integer and >1) and 

10 fiB.vB 

£ Ifm^-e 2 *"""^*"""' (6) 

M and A/ are additional, not-necessarily integer parameters which correspond to the actual 
position of the center of the sideband, as discussed in E. Voelkl, L. F. Allard, B. Frost, 

15 "Practical Electron Holography: Applications of Advanced Hologram Processing Techniques 
to Materials Science Problems", Electron Holography. Eds. A. Tonomura, L. F. Allard, G. 
Pozzi, D. C. Joy, Y. A. Ono, 1995 Elsevier Science B.V. The effect of &k and A/ is to move 
the center of the sideband to the origin in reciprocal, or Fourier, space. However, while in 
"Practical Electron Holography: Applications of Advanced Hologram Processing Techniques 

20 to Materials Science Problems", supra , the range for K and V was given as being identical to 
the range for k and 1 of the standard Fourier transform, the range for k and /' is significantly 
reduced in accordance with the inventive arrangements, thus reducing computational time and 
memory requirements. Moreover, these reductions are possible without loss of information, 
as the actual sideband necessary for the reconstruction process contains significantly fewer 
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pixels than the full Fourier transformation (see, for example "Optimized Sampling Schemes 
For Off- Axis Holography", supra) . The allowed values for K and V in accordance with the 
inventive arrangements, namely, 

*' e {-ICI2 y ...JCI2-\ } and / e {-Z.72,...,L72-I} (7) 

are a subset of the allowed values for k and /, where fC and V are integers and even with 1 < 
K* < Mil and 1 < V < Nil. For example, for the special case where M and N are integers and 
of the power of two, A7 and L can be conveniently selected as 

K' = M/8 and V = M/8, thus reducing the number of elements of G' to be computed by a 
factor of 16. 

G(k\F) can be computed in three ways, either by using Equation (3) directly, or by 
using Equation (4), where the real or integer image is first converted to a complex data type 
and then transformed using the reduced k' and V values, or by using the approximation in 
Equation (5), where the real or integer image is first converted to a complex data type, then 
binned, and then transformed using the standard Fourier transform. 

The not necessarily integer numbers Ak and A/ need to be predetermined. This is can 
be easily accomplished using one of many methods described in the literature and need not be 
discussed here. For most applications and in practice, the values of A£ and A/ remain 
unchanged, or assume only values that are already known. 

After the application of the FFT and isolation of the sideband in method JO, and after 
application of only the EFT in method 40, the remaining steps are the same, it being 
understood that the method 40 according to the inventive arrangements can proceed in 
substantially real-time, whereas the prior art method can not. 

An aperture is applied to separate the sideband from the autocorrelation represented 
by the application of the FFT and EFT respectively, in accordance with steps 20 and 50. An 
inverse Fourier transform is applied in accordance with steps 22 and 52 to return to real space 
with a complex image, that is, an image having both phase and amplitude information. 
Finally, an amplitude or phase image, or both, is extracted in accordance with steps 24 and 
54. 

The centering of the sideband, as necessary for the conventional reconstruction 
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method, is unnecessary in accordance with the inventive arrangements, as the EFT 
automatically delivers the sideband centered. This has a significant implication in further 
reducing the computational time for the reconstruction process. 

Many modern digital cameras allow binning of the image data by the camera 
hardware, where binning takes either no time or insignificant time. As a matter of fact, the 
readout time of an image from a camera with hardware binning is significantly reduced, 
allowing for more frames per second for binned images than can be reached for non-binned 
images. Binning of images, for example by a factor of two, means that a 2 x 2 pixel area is 
treated as one pixel. Thus a 1024 x 1024 pixel image shrinks to 512 x 512 pixels, or 1/4 of 
the original number of pixels. 

For any holography or interferometry involving a constant carrier frequency, the 
consequences of binning are that the sampling frequency of the carrier frequency falls either 
very close to, on, or even beyond, the Nyquist limit. 

The inventive arrangements taught herein can be used for holograms or interferograms 
or binned versions thereof such that the carrier frequency falls either close to or beyond the 
Nyquist limit. The effect of using binning in addition to using Equations (3), (4) will shorten 
the reconstruction time by an additional factor of almost 4. 

The invention described here has several advantages over the conventional 
reconstruction methods of the prior art. The time for computing G is significantly shorter 
than for computing G. The memory requirements to compute G over G are reduced; thus 
allowing fast parallel processor boards with a limited amount of memory to work on larger 
image formats. The sideband is automatically centered. 

The inventive arrangements significantly advance the field of off-axis holography and 
interferometry by making reconstructed phase and/or amplitude images available 
immediately. The inventive arrangements allow investigation of a sample in real time with 
holography and/or interferometry tools because feedback is immediate. Immediate feedback 
is not presently possible because reconstruction times are almost two orders of magnitude too 
long. The inventive arrangements support research where information on phase shifts of 
imaging waves is of interest. For example, in electron holography for materials 
characterization, where the interest is for example in the evaluation of magnetic and electric 
fields in non-volatile memory components, a researcher can examine the important areas of 
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the sample with much greater facility. The inventive arrangements also allow one to make 
dynamic, near real-time observations to determine the effect of changes of the fields. The 
inventive arrangements support the acquisition and evaluation of surface profile information, 
morphology information, information on small (for example, less than 1 wavelength) 
perturbations of surfaces and depth information on almost any scale. For example, using 
optical holography, the fingertip morphology could be recorded as a 3-dimensional pattern, 
allowing 3-dimensional fingerprints to be stored and subsequently compared in real-time for 
personal identification purposes, making forgery of fingerprints significantly more difficult. 

Object recognition can be enhanced by optical holography, which will allow a 3- 
dimensional surface profile to be recorded and processed at reasonable speed, thus replacing 
the present technology which uses a flat, 2-dimensional image. It is expected that most areas 
of present imaging interferometry or holography will benefit from the rapid data evaluation 
provided by the inventive arrangements. New application areas for 
interferometry/holography can be initiated, as the main obstacle of time consuming 
reconstruction procedures is removed. 
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What is claimed is: 

1 . A method for reconstructing a digitized image, comprising the steps of: 
retrieving from memory a digitized image having parameters defining a position of a 

sideband of said image in Fourier space; 

applying the Extended Fourier transform to said image function to directly and 
correctly center said sideband; and, 

displaying a resulting image in at least one of phase and amplitude. 

2. The method of claim 1, comprising the step of retrieving from said memory a 
digitized image of an off-axis hologram. 

3. The method of claim 1, comprising the step of retrieving from said memory a 
digitized image of an off-axis interferogram. 

4. The method of claim 1, comprising the step of retrieving from said memory a 
digitized image originating with a digital camera. 

5. The method of claim 1, comprising the step of retrieving from said memory a 
digitized binned image originating with a digital camera. 

6. The method of claim 1, comprising the steps of: 
first digitizing an image of an off-axis hologram; 
defining the parameters; and, 

storing the parameters in said memory. 

7. The method of claim 1 , comprising the steps of: 

first digitizing an image of said off-axis interferogram; 
defining said digitized image by the parameters; and, 
storing said image function in said memory. 
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8. The method of claim 1, comprising the steps of: 
binning an image originating with a digital camera; 
defining the parameters; and, 

storing the parameters in said memory. 

9. The method of claim 1 , wherein said steps are undertaken independently of 
wavelength. 

10. The method of claim 1, wherein said steps are undertaken independently of 
wave type. 

11. The method of claim 10, wherein said steps are undertaken independently of 
wavelength. 

12. The method of claim 1, implemented substantially in real-time. 

1 3. The method of claim 1 , comprising the steps of: 
applying an aperture to limit said sideband; and, 

applying an inverse Fourier transform to said aperture limited sideband 

14. The method of claim 1 , wherein the digitized image is defined by an image 
function, the image function having the parameters defining a position of the sideband of the 
image in Fourier space. 

15. The method of claim 14, comprising the steps of: 
first digitizing an image of an off-axis hologram; 
defining said digitized image by said image functions; and, 
storing said image function in said memory. 
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16. The method of claim 15, comprising the step of including in said image 
function terms representing image amplitude, image contrast at the interference fringes, 
image phase and pixel sampling frequency. 

17. The method of claim 14, comprising the steps of: 
first digitizing an image of said off-axis interferogram; 
defining said digitized image by said image function; and, 
storing said image function in said memory. 

18. The method of claim 17, comprising the step of including in said image 
function terms representing image amplitude, image contrast at the interference fringes, 
image phase and pixel sampling frequency. 

19. The method of claim 14, comprising the steps of: 

first digitizing a binned image originating with a digital camera; 
defining said digitized image by said image function; and, 
storing said image function in said memory. 

20. The method of claim 19, comprising the step of including in said image 
function terms representing image amplitude, image contrast at the interference fringes, 
image phase and pixel sampling frequency. 

21 . A method for reconstructing a digitized image, comprising the steps of: 
retrieving from a memory a digitized image of at least one of an off-axis 

hologram, an off-axis interferogram and a binned image originating with a digital camera, 
said digitized image having parameters defining a position of a sideband of said image in 
Fourier space; 

applying the Extended Fourier transform to said image function to directly and 
correctly center said sideband; and, 

displaying a resulting image in at least one of phase and amplitude. 
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22. The method of claim 21, comprising the steps of: 
first digitizing said image; 

defining the parameters; and, 

storing said image function in said memory. 

23. The method of claim 21, wherein said steps are undertaken independently of 
wave type and independently of wavelength. 

24. The method of claim 21, implemented substantially in real-time. 

25. The method of claim 2 1 , comprising the steps of: 
applying an aperture to limit said sideband; and, 

applying an inverse Fourier transform to said aperture limited sideband; 
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